Time-Contrastive Networks: Self-Supervised Learning from Video

We propose a self-supervised approach for learning representations and robotic behaviors entirely from unlabeled videos recorded from multiple viewpoints, and study how this representation can be used in two robotic imitation settings: imitating object interactions from videos of humans, and imitating human poses. Imitation of human behavior requires a viewpoint-invariant representation that captures the relationships between end-effectors (hands or robot grippers) and the environment, object attributes, and body pose. We train our representations using a metric learning loss, where multiple simultaneous viewpoints of the same observation are attracted in the embedding space, while being repelled from temporal neighbors which are often visually similar but functionally different. In other words, the model simultaneously learns to recognize what is common between different-looking images, and what is different between similar-looking images. This signal causes our model to discover attributes that do not change across viewpoint, but do change across time, while ignoring nuisance variables such as occlusions, motion blur, lighting and background. We demonstrate that this representation can be used by a robot to directly mimic human poses without an explicit correspondence, and that it can be used as a reward function within a reinforcement learning algorithm. While representations are learned from an unlabeled collection of task-related videos, robot behaviors such as pouring are learned by watching a single 3rd-person demonstration by a human. Reward functions obtained by following the human demonstrations under the learned representation enable efficient reinforcement learning that is practical for real-world robotic systems. Video results, open-source code and dataset are available at https://sermanet.github.io/imitat

The Murchison Widefield Array: Design Overview

The Murchison Widefield Array (MWA) is a dipole-based aperture array synthesis telescope designed to operate in the 80-300 MHz frequency range. It is capable of a wide range of science investigations, but is initially focused on three key science projects. These are detection and characterization of 3-dimensional brightness temperature fluctuations in the 21cm line of neutral hydrogen during the Epoch of Reionization (EoR) at redshifts from 6 to 10, solar imaging and remote sensing of the inner heliosphere via propagation effects on signals from distant background sources,and high-sensitivity exploration of the variable radio sky. The array design features 8192 dual-polarization broad-band active dipoles, arranged into 512 tiles comprising 16 dipoles each. The tiles are quasi-randomly distributed over an aperture 1.5km in diameter, with a small number of outliers extending to 3km. All tile-tile baselines are correlated in custom FPGA-based hardware, yielding a Nyquist-sampled instantaneous monochromatic uv coverage and unprecedented point spread function (PSF) quality. The correlated data are calibrated in real time using novel position-dependent self-calibration algorithms. The array is located in the Murchison region of outback Western Australia. This region is characterized by extremely low population density and a superbly radio-quiet environment,allowing full exploitation of the instrumental capabilities.Comment: 9 pages, 5 figures, 1 table. Accepted for publication in Proceedings of the IEE

A new layout optimization technique for interferometric arrays, applied to the MWA

Antenna layout is an important design consideration for radio interferometers because it determines the quality of the snapshot point spread function (PSF, or array beam). This is particularly true for experiments targeting the 21 cm Epoch of Reionization signal as the quality of the foreground subtraction depends directly on the spatial dynamic range and thus the smoothness of the baseline distribution. Nearly all sites have constraints on where antennas can be placed---even at the remote Australian location of the MWA (Murchison Widefield Array) there are rock outcrops, flood zones, heritages areas, emergency runways and trees. These exclusion areas can introduce spatial structure into the baseline distribution that enhance the PSF sidelobes and reduce the angular dynamic range. In this paper we present a new method of constrained antenna placement that reduces the spatial structure in the baseline distribution. This method not only outperforms random placement algorithms that avoid exclusion zones, but surprisingly outperforms random placement algorithms without constraints to provide what we believe are the smoothest constrained baseline distributions developed to date. We use our new algorithm to determine antenna placements for the originally planned MWA, and present the antenna locations, baseline distribution, and snapshot PSF for this array choice.Comment: 12 pages, 6 figures, 1 table. Accepted for publication in MNRA